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ABSTRACT 

Faculty at Rice University are creating instructional resources to support teaching first-year 
engineering design using a flipped classroom model. This implementation of flipped pedagogy is 
unusual because content-driven, lecture courses are usually targeted for flipping, not project-based 
design courses that already incorporate an abundance of active learning. However, during the first 
five semesters in which first-year engineering design was offered at Rice, almost 30% of class time 
conformed to the traditional lecture model. In fall 2014 a partially flipped model that placed greater 
emphasis on higher levels of Bloom’s taxonomy during class time was introduced to facilitate student 
development in design topics. To achieve this goal, lecture time was replaced with in-class exercises 
that require students to analyze and evaluate design situations or problems, many of which were 
carefully crafted to expose common pitfalls that occur during the design process. 

To date, the team has produced flipped classroom resources for ten modules of the engineering 
design process and professional skills: design criteria, user-defined scales, pairwise comparison 
charts, brainstorming, decomposition, morphological charts, Pugh screening matrix, Pugh scoring 
matrix, Gantt charts, and presenting a design proposal. Each module includes three components: 
topical videos, summative quizzes, and in-class exercises. 

Work is ongoing to examine the impact of using a flipped classroom model in this first-year 
engineering design course. Two assessment methods have been deployed, and a third one is 
underway. One assessment method uses a pre- and post-course assignment to measure students’ 
application of the design process. A second method focuses on student exam scores. A direct com¬ 
parison of student learning in the partially flipped model versus the lecture model shows no statisti¬ 
cally significant differences, which is consistent with some implementations reported in the literature. 

Key words: Flipped classroom, engineering design, first-year, Gantt chart 
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BACKGROUND 

Emergence of Flipped Classroom as a Popular, Disruptive Learning Model 

The flipped classroom model has become a buzzword in the popular press for a number of reasons, 
including the drive toward more active classrooms and the model’s pervasive use of technology [1] 
[2] [3] [4]. This novel pedagogical approach transforms educational delivery by inverting the tradi¬ 
tional workflow of each lesson, placing the lecture out of class and the activities or the application 
of conceptual knowledge in class. The result is that the highest cognitive load material is shifted into 
the classroom where the instructor is present to address student-specific needs. Another advantage 
of this model is that it engages students through technology, which is a standard part of their daily 
lives. To the general public this model is perceived as one of the most transformative developments 
in teaching in the modern age. To education specialists, this model is neither radical nor the panacea 
to institutional education, but the model’s emergence and popularity support the evolution of and 
testing of a number of active learning techniques [4]. 

Though the flipped classroom may be viewed as a singular educational model, it serves as an 
overarching umbrella under which multiple educational models may be employed. The flipped 
classroom can be unpacked into two components: outside the classroom and inside the class¬ 
room. The first component may be reductively described as replacing didactic lectures with 
rich media including but not limited to videos, podcasts, web applets, and static documents, 
which students view outside of class. Eliminating content-based lectures allows instructors to 
reallocate class time for research-based, active learning techniques such as question and answer 
periods, group problem solving, think-pair-share, cooperative learning, peer-to-peer learning, 
and small group discussion [4] [5] [6] [7] [8]. These techniques are shown to enhance student 
engagement and cognition. 

Contributing Educational Models 

The motivation for flipping a classroom is often based on a desire to increase student-centered 
learning during the class period via active learning strategies [4] [8]. These strategies are governed by 
well-established educational models developed in the previous century by a number of psychologists, 
educators, and philosophers and more recently adapted by engineering educators. In the context 
of teaching engineering design, contributing models include constructivism, cooperative learning, 
and the zone of proximal development. 

In the constructivist model, students assimilate new information with their prior knowledge, 
experiences, beliefs, and values to construct a representation of reality [7]. Constructivist 
pedagogy posits a more student-centered approach and attempts to target instruction within a 
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student’s zone of proximal development [9]. In the zone of proximal development students are 
pushed to extend their capabilities through interactions with others who possess more knowledge 
or experience. In a classroom setting, this expert guidance may come from professors or peers 
[7] [10]. For example, in peer-to-peer learning, small groups can practice a new skill to increase 
competency and to learn from others’ mistakes in a low stakes situation before they tackle their 
own projects [11]. 

This type of cooperative learning is frequently leveraged as a pedagogy of engagement in the 
flipped classroom, and in engineering education more generally [12]. Students work interdepen¬ 
dent^ in small groups to accomplish a common goal, such as completing a design challenge. Smith 
et al.’s survey of evidence-based research on cooperative pedagogy indicates that it generally 
has a positive effect on success in college, critical thinking, peer relationships, and motivation to 
study engineering. 

Assessment of the Flipped Classroom to Improve Learning Outcomes 

The flipped classroom model was developed and first implemented in K-12 STEM educational 
settings; however, faculty at universities have begun to flip their classrooms as well. Many edu¬ 
cators would like to know how implementations of the flipped model affect student attitudes 
and learning outcomes. Two key review papers synthesize results derived from a broad range of 
flipped courses at multiple universities. Bishop and Verleger provide a valuable overview of as¬ 
sessment projects within the flipped classroom community, carefully distinguishing between fully 
and partially flipped deployments of the model [4]. Phillips and O’Flaherty’s review of flipped 
classroom implementations describes the type of class, the assessments completed, and the 
outcomes [8]. Velegol et al.’s recent paper also presents a useful summary table that classifies 
flipped implementations [13]. 

Qualitative assessment of the flipped classroom model features prominently in the literature as a 
measure of success. Students’ opinions vary widely. They tend to respond favorably to the increased 
engagement these courses foster, but some respond negatively to the increased personal respon¬ 
sibility for their own learning outside of the course [14] [15] [16] [17] [18]. Additional qualitative 
feedback suggests that the flipped classroom offers opportunities to develop professional skills such 
as communication and teaming [19] [20] [21] [22] [23]. Kim et al.’s paper analyzes the results from 
three simultaneously flipped classrooms in engineering, sociology, and humanities, which they use 
as the basis for defining a set of curriculum design principles for flipped courses [24]. 

When considering assessment of learning outcomes, most research on flipped classroom models 
compares a content-rich, didactic lecture course to a flipped version. Student performance is typically 
assessed using pre- and post-tests of content-based knowledge, exams, or course grades [13] [19] 
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[25] [26] [27] [28] [29] [30] [31]. The results of these studies and others are mixed with respect to 
improvements in student performance, regardless of whether a course was fully or partially flipped. 
For example, a flipped human-computer interaction course demonstrated improvement in student 
grades for homeworks, projects, exams and final course grades as compared with a control group 
[32]. In other courses though, it is less clear that the flipped model contributed to an improved 
classroom environment. In a fluids course at the U.S. Military Academy no differences in performance 
were observed between the treatment and control groups [33]. 

Flipping First-Year Courses 

In contrast to content-rich courses that focus on delivery of theories and principles, engineering 
design and other project-based courses usually emphasize process knowledge. Process knowledge, 
or ‘how to do’ something, is essential in teaching problem solving, the design process, modeling, 
and research. Examples of engineering process-focused courses include senior capstone design, 
first-year design, and research-based courses. 

Several studies report on efforts to flip first-year engineering design courses or introductory 
engineering courses that target first-year students. Assessments of these flipped courses are 
primarily based on survey responses that capture students’ general attitudes about the flipped 
classroom experience and students’ use of and satisfaction with newly deployed learning materials 
that support flipped instruction. Survey data are sometimes supplemented with course evalua¬ 
tion data. Relatively few studies provide evidence of the effects of the flipped model on student 
learning outcomes. 

Many studies have found that students respond positively to the use of flipped instructional 
materials such as e-books, videos, quizzes, podcasts, demos, and active learning exercises [34]. 
Rowan University introduced a PathFinder e-book paired with before/after reading exercises as part 
of its flipped implementation and reported that students and faculty responded positively to these 
materials [35]. Students especially liked getting immediate feedback on the reading exercises to 
assess their knowledge and comprehension. Schluterman et al. implemented videos and quizzes in a 
first-year Introduction to Engineering course at the University of Arkansas and found that students 
slightly preferred video content over lecture [36]. 

Engineering educators also report that first-year students either respond positively to the overall 
effectiveness of the flipped model from the outset or express a greater affinity for it over time 
relative to traditional modes of instruction [35]. Reidsema and Kavanagh reported that 73% of the 
students in a large-scale, first-year engineering design course at the University of Queensland in 
Australia rated the flipped course “satisfactory or better” in its first year of implementation [37]. 
The University of Maryland piloted a flipped version of an established, large-scale Introduction to 
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Engineering Design course for first-year students. Students’ mean rating of the overall effective¬ 
ness of the flipped lecture model was 4.35 (5 is “very effective”) compared to students’ mean 
rating of 3.63 for the traditional lecture model (P<0.001). In fact, the flipped course was rated 
higher than the lecture course on multiple dimensions, which led the faculty to flip all four sec¬ 
tions of the course [34]. 

While the body of literature on implementations of flipped first-year engineering courses is 
growing, less is known about how these implementations affect learning outcomes. Those that 
report on learning outcomes primarily rely on students’ quiz grades, problem sets, and exam 
scores to support claims about impact. A flipped first-year engineering honors course at Ohio 
State University, which incorporated videos, quizzes, readings, and in-class activities designed 
to move students to the higher levels of Bloom’s taxonomy, showed no statistically significant 
change in students’ exam scores [38]. Maarek & Kay flipped Introduction to Biomedical Engi¬ 
neering, a course for first-year students at the University of Southern California. They compared 
students’ scores on final exams in the flipped course to exam scores from the previous year when 
it was taught as a lecture course. The average exam scores in each were 16.6 ± 2.6 and 18.1 ± 1.7 
out of 20 (P=0.001) respectively, which indicates the flipped course negatively impacted learn¬ 
ing outcomes [39]. 

To supplement the literature on flipped first-year engineering design courses, we examined evi¬ 
dence of learning outcomes in other design courses that have been flipped [40] [41]. Merrett, for 
example, measured his students’ performance on exams in sophomore and senior level mechanical 
engineering design courses. He compared students’ scores on exams in courses that combined case- 
based pedagogy with traditional lectures to flipped versions of the courses that combined cases with 
either assigned reading in a textbook or videos. He found that the use of videos produced a statisti¬ 
cally significant improvement in students’ performance on the final exam compared to traditional 
lectures, whereas the flipped version of the courses with assigned readings had no statistically 
significant effect or a weak effect on students’ exam grades [42]. 

Given this context, our paper’s emphasis on learning outcomes provides a timely contri¬ 
bution to the scholarship on the flipped classroom model. Our first-year engineering design 
course represents a relatively unique instantiation of this pedagogical approach because the 
course contained significant active learning prior to flipping the lecture content. Its emphasis 
on process knowledge, or “how to do” engineering design, is distinct yet widely applicable, as 
design is the centerpiece of many first-year introductory engineering courses. In addition, the 
focus of this manuscript is the assessment of student performance regarding design process 
knowledge, rather than students’ perceptions of the teaching methodology or the quality of the 
flipped instructional materials. 
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DESCRIPTION OF FIRST-YEAR DESIGN COURSE 

Introduction to Engineering Design-Course Overview 

Introduction to Engineering Design (ENG1120) is a one-semester multidisciplinary design course 
at Rice University. The course is an elective course available for all first-year students in the School 
of Engineering. It was offered for the first time in spring 2011 and has been offered every semester 
since. At its core, ENG1120 is a project-based course built on best practices that emphasize student- 
centered, active, cooperative learning. Teams of students work for an entire semester on an authentic, 
open-ended design challenge and produce a physical prototype for their client. 

The course outcomes, structure, and deliverables are described in detail elsewhere [43] [44]. In 
brief, three specific learning outcomes guide the course design and deliverables: 

1. Students design a product that meets a user-defined need and realistic constraints. Specifi¬ 
cally, students develop realistic design criteria, apply appropriate methods for brainstorming 
to generate multiple design solutions, use decision matrices to select among design solution 
options, and iteratively prototype a physical product. 

2. Students effectively communicate progress of their design using written and oral/visual 
communication. 

3. Students function effectively on a high-performance team and use project management tools 
to guide the team’s work. 

The design of the course is informed by Vygotsky’s theoretical work on zones of proximal devel¬ 
opment [9]. Based on his observations of children learning a task, Vygotsky concluded that children 
were capable of performing tasks that exceeded their individual abilities if they were guided by 
individuals with more expertise. In the context of ENG1120, teams benefit from regular contact with 
an instructional support team, which includes course faculty, faculty mentors with relevant technical 
expertise, teaching assistants, writing mentors, and shop technicians. These individuals coach the 
design teams by sharing their expertise, modeling more rigorous, systematic thinking, and providing 
formative feedback. This type of guided discovery or cooperative learning allows instruction and 
feedback to be tailored to the needs of individual students and teams. It also mitigates the cognitive 
overload many students experience when they simultaneously attempt to acquire new knowledge 
and engage in complex problem-solving [11]. 

Learning is not only promoted through the instructional support team, but also through care¬ 
fully structured course materials. Students engage in in-class activities and assignments related to 
the design process, which stimulate creative and critical thinking. These materials also challenge 
students at the appropriate level to master concepts, which are fundamental to the design process 
[45]. During class, students construct and apply their increasingly complex understanding of these 
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concepts to their design projects [7]. Because design is an iterative process, students get multiple 
opportunities to practice testing and evaluating potential solutions throughout the course, which 
is essential [7] [45]. 

Established metrics are used to assess students’ design knowledge, communication skills and 
team skills throughout ENGI 120. Deliverables include nine technical memos and a final report to 
communicate decisions at each step of the design process [44]. Each team gives two 15-minute 
oral presentations and undergoes two formative prototype checks and one final graded prototype 
evaluation. An end-of-course exam covers the application of the engineering design process and 
professional skills. Team skills are assessed through CATME [46]. 

The design challenges students tackle are drawn from local health care providers, community 
partners, industry, humanitarian organizations, and Rice University. Faculty members scope and 
define the design challenges. Students are placed on teams of 4-6 students based on their interests, 
with each team working on a different client-sponsored project (Table 1). 

The course deploys an engineering design framework that encompasses a design analysis stage 
and a solution stage (Figure 1). The first half of the semester is devoted to clarifying the design 
problem, developing the design context review, establishing design criteria, brainstorming solutions, 
using a Pugh matrix to evaluate and select a solution, and then defining the selected solution in 
detail. During the second half of the semester, student teams build and test a series of prototypes. 

ENG1120 is taught in the Oshman Engineering Design Kitchen (OEDK), an innovative engineering 
design facility that features a traditional classroom and a large open workspace with co-located 
tables. This collaborative learning environment facilitates team interactions as well as student-faculty 
interactions, providing ample opportunity for just-in-time instruction that addresses immediate 
problems when teams are most receptive to input [11]. In addition, the OEDK offers a vast array of 


Project topic 

Client/Sponsor 

Medically-motivated 

Forearm rotation measurement 
Numbing agent for needle injection 
Tracheostomy tube mannequin 

Physician at Shriner’s Hospital 

Physician at Texas Heart Institute 

Physician at Texas Children’s Hospital 

Global reach 

Window design for schools 

Physical therapy aid 

Blood pressure calibration 

Project Schoolhouse 

Harmony Children’s Disability Ctr, Buduburam, Ghana 
Biomedical engineer, Addis Ababa, Ethiopia 

Local reach 

Baby bird brooder box 

Robot obstacle course 

Campus aid for visually impaired 

Houston Zoo 

NASA 

Rice University Disability Support Services 


Table 1. Sample projects in ENG1120 (fall 2014). 
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Engineering Design Process 



Figure 1. Engineering Design Process. 


hand tools, prototyping supplies, and several pieces of advanced manufacturing equipment, which 
teams use to construct design solutions [47]. 

Class Time in ENGI 120 before Introduction of Flipped Classroom Model 

ENGI 120 is taught on Tuesday and Thursday in 75-minute class periods. Figure 2 illustrates how 
class time is allocated in ENGI 120. For the 28 instructional days, 16 focus on a single technical or 
professional topic that encompasses the entire 75 minutes. Introductory material and class oral 
presentations account for six days (22% of class time); six additional days are dedicated entirely 
to team meetings. 

For those remaining 16 instructional days, the 75-minute period is split into two halves. In the 
first half, the class assembles to learn about design process knowledge or professional skills. These 
two content blocks are each 14%. In the remaining half of these 16 class periods, the teams meet to 
work on their projects. Overall, 50% of class is dedicated to team meetings. 

From spring 2011 through spring 2014, ENGI 120 was taught using a lecture/team meeting model 
(referred to as “lecture”)- When we employed the lecture model, the course instructors delivered 
PowerPoint lectures during the 16 class periods that covered the engineering design process and 
professional skills. 

Rationale for Flipped Model in ENG1120 

As instructors, we observed that students struggled to move from hearing about a topic during 
the first 30 minutes of class to applying it to their design project immediately thereafter. This 
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Introduction 

(2 full class periods) 


Oral Presentations 

(4 full class periods) 

Team time 

(6 full class periods, 

16 half-class periods) 

Design Process 
Knowledge 

(8 half-class periods) 

Professional Skills 

(8 half-class periods) 



Figure 2. Time breakdown for ENG! 120. 


pedagogical approach required students to jump directly from the lowest levels of Bloom’s taxonomy, 
remembering and understanding material presented in a PowerPoint lecture, to the highest levels 
of the taxonomy, evaluating and synthesizing in their design team. 

Upon reflection about how better to support student learning in light of Bloom’s taxonomy, we 
implemented a flipped classroom model in which pedagogical material targeted tiered, discrete 
learning stages. Specifically, students watched videos and took quizzes prior to coming to class to 
master key vocabulary, concepts and processes. During class small groups of students applied their 
knowledge to solve short in-class exercises. Then, the design teams met to tackle their own design 
problem, a high-level and challenging task. 

Class Time in ENGI 120 after the Introduction of Flipped Classroom Model 

Starting in fall 2014, the authors began incorporating the flipped/team meeting model. The class 
still met together for the first half of those 16 class periods, but instead of listening to the instructor 
lecture, students completed in-class exercises (ICE) in which they practiced a step in the design 
process or a professional skill. 

Figure 3 illustrates the transition of the lecture to the flipped model. In spring 2014 and prior, 
the eight design process and eight professional skills lectures were all taught in the lecture model 
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(Figure 3A). In fall 2014, four of the eight lecture periods on the design process were flipped (re¬ 
ferred to as “partially flipped”) (Figure 3B). Each lecture was replaced with ICEs. In spring 2015, 
two professional skills ICEs were added (Figure 3C). By fall 2015, all eight design process lectures 
and six professional skills lectures were transitioned from lecture to ICEs (Figure 3D). The time 
set aside exclusively for the introduction to the course, team time, and oral presentations remains 
unchanged (Figure 2). 


VIDEOS, QUIZZES AND IN-CLASS EXERCISES COMPRISE 
THE FLIPPED CLASSROOM MATERIALS 

By spring 2015, flipped classroom materials for six topics on the engineering design process and 
professional skills had been produced (Table 2). For each topic, there are one to three modules. The 
ten implemented modules include design criteria, user-defined scales, pairwise comparison charts, 
brainstorming, decomposition, morphological charts, Pugh screening matrix, Pugh scoring matrix, 
Gantt charts, and presenting a design proposal. Table 3 shows topics and modules that have been 
developed since spring 2015. For each module, the instructional materials include videos, quizzes, 
and ICEs. 

Videos 

The purpose of the videos is to shift transmission of key content from the classroom lecture to 
rich media that can be watched outside of the classroom. The videos are 1-11 minutes in length, 
except for the video on presenting a design proposal, which is about 1 hour in length. As shown 
in Tables 2 and 3, each module includes one to five instructor videos and up to three student 
team videos. The instructor videos are a mixture of one of the instructors talking, the instructors 
dialoguing with each other, slide-show presentations (e.g., PowerPoint), and an instructor talking 
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Module Name 

# of Instructor 
Videos 

# of Design 
Team Videos 

Playlist Link 

Topic: Design Criteria [F14, S15] 

Design criteria 

3 

3 

https://goo.gl/SrIh2K 

User-defined scales 

1 

2 

https ://goo. gl/lTL JFR 

Pairwise comparison charts 

3 

3 

https://goo.gl/Q3J17X 

Topic: Brainstorming [F14, S15] 

Brainstorming 

3 

2 

https ://goo. gl/nb4DsN 

Topic: Engineering Decision Making I [F14, S15] 

Decomposition 

1 

2 

https ://goo. gl/aW 8h5f 

Morphological charts 

1 

3 

https://goo.gl/2Uv2tv 

Pugh screening matrix 

5 

2 

https://goo.gl/Yeq4GN 

Topic: Engineering Decision Making II [F14, S15] 

Pugh scoring matrix 

2 

2 

https://goo.gl/yshOVC 

Topic: Project Planning [S15] 

Gantt charts 

3 

0 

https://goo.gl/QUY Q5x 

Topic: Presenting a Design Proposal [S15] 

Presenting a design proposal 

1 

0 

https://goo.gl/5gWCDW 


Table 2. Details of video contents used in Fall 2014 [F14] and/or Spring 2015 [S15J. For 
each topic, which aligns with a class day, there may be one or more modules. Playlist links 
are live. 


accompanied by text. Instructor videos focus on describing methods, defining relevant terms, 
and explaining strategies. 

Most modules also feature student design team videos. These videos present a reenactment 
of a former ENGI 120 design team tackling its previous design problem at that step of the en¬ 
gineering design process. Three former teams were selected to appear in the videos to create 
consistency across the series with respect to demonstrated projects. Student videos are scripted 
to illustrate common pitfalls and best practices. More detailed descriptions of the instructor 
and student videos can be found elsewhere [48]. Most of these playlists can be found online at 
http://goo.gl/dPMdaO. 

Quizzes 

Online quizzes monitor students’ acquisition of knowledge, understanding, and application of 
the key terms and processes presented in the instructor videos. For each module, ten to twenty-five 
questions have been developed. The quizzes are hosted on http://goo.gl/A7cK4S. See Appendix A 
for sample quiz questions. 
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Module Name 

# of Instructor 
Videos 

# of Design 
Team Videos 

Playlist Link 

Topic: Engineering Design Process Overview 

EDP Overview 

1 

0 

http s ://goo. gl/xnOfDp 

Topic: Clarify Team Assignment 

Clarifying team assignments 

1 

3 

https://goo.gl/sXT201 

Topic: Understanding Problem and Context 

Understanding the problem and context 

5 

2 

https://goo.gl/wiOeJy 

Topic: Prototyping 

Overview 

2 

0 

http s ://goo. gl/f40fJB 

Tools 

1 

0 

http s ://goo. gl/xz4e01 

Low and medium fidelity prototypes 

2 

4 

https://goo.gl/zpTvWS 

Topic: Testing 

Testing 

4 

0 

https://goo.gl/05hCBC 

Topic: Teaming 

Teamwork 

3 

0 

https://goo.gl/E716Sz 

Topic: Intellectual Property 

Intellectual property 

1 

0 

http s ://goo. gl/vRoj 8B 


Table 3. Details of video contents developed after Spring 2015. For each topic, which 


aligns with a class day , there may be one or more modules. Playlist links are live. 


In-class Exercises 

In-class exercises strengthen students’ understanding of the design process by requiring them 
to practice the steps in the engineering design process and their professional skills prior to ap¬ 
plying them to their own design project. The ICEs prompt students to “apply problem-solving 
heuristics appropriate to the domain” that focus on a specific goal [11] [45]. Examples include 
applying knowledge to a new problem, evaluating a completed design scenario, and applying 
the design process to a team’s specific project. Prior to fall 2014, we had developed 30 ICEs, ap¬ 
proximately three for each module. None of the ICEs are specific to the ongoing semester-long 
design projects. The in-class exercises are hosted on http://goo.gl/A7cK4S. See Appendix A for 
sample ICEs. 

Production of Flipped Materials 

The course faculty have invested significant time and financial resources in creating rich me¬ 
dia instructional materials for first-year engineering design. In our experience, approximately 
10-20 hours of time over a period of two months was required to produce 30-45 minutes of 
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high-quality video. Individuals involved in producing the videos included the instructional faculty, 
videographer, project manager, and video editor. In contrast, creating the in-class materials and 
quizzes was relatively straightforward. This task was accomplished by the instructional faculty 
and upper-class student assistants. 

Producing materials for the flipped course forced us to think more deeply about the process 
knowledge we were delivering. For example, to create video storyboards, we essentially reworked 
most of the “lecture” content from scratch, which required us to identify the most salient points to 
highlight in the videos. This discussion was particularly productive in terms of defining the steps 
necessary to create the charts taught in the class (e.g., Pugh matrix, Gantt chart, pairwise compari¬ 
son chart). Furthermore, creating these videos meant acquiring proficiencies in areas not normally 
associated with curriculum design, such as script writing, storyboarding, video production, editing, 
and animating illustrations. 

Implementation of Flipped Model Materials 

The videos and quizzes were integrated into a university course management system. The videos 
were also hosted on YouTube as playlists that allow access to analytics that measure the extent to 
which students are watching the videos. To implement the flipped module, an assignment that cov¬ 
ers one topic (and may include one or more modules) was initiated 48-72 hours prior to the class 
period. To prepare for a flipped class day, students watched several videos and completed one quiz 
for each module. 

During the fall 2014 implementation, students were expected to spend 12-42 minutes per topic 
on the instructor videos. An average watch time of instructor videos was 23 minutes per class. 
For each topic, students were expected to watch one or more student team videos, which added 
5-10 min per team video. For the partially flipped model in fall 2014, students were required to watch 
videos prior to coming to class for four of the eight class periods that focused on the engineering 
design process. At full implementation, about half of the class periods require watching videos of 
comparable lengths. 

By administering the quizzes online before class, instructors were able to see which students 
took the quizzes, when, and which questions were answered incorrectly. This allows us to modify 
the ICEs and/or discuss misconceptions or gaps in students’ knowledge during class time. 

During the first half of class, students assemble in small groups of two or three (not their team) 
and work through two ungraded ICEs related to the topic. Faculty and teaching assistants circulate 
to provide assistance. Faculty occasionally call the class together to discuss key points or to 
encourage reflection on the process. For the second half of the class, students disperse into their 
design project teams. 
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ASSESSMENT OF THE IMPACT OF TRANSITIONING FROM LECTURE TO PARTIALLY FLIPPED MODEL 

As stated earlier, the first learning outcome for ENGI 120 is for students to design a product that 
meets a user-defined need and realistic constraints. In order to measure this learning outcome, a 
method to assess design process knowledge must be selected. A variety of quantitative, qualitative, 
and mixed methods have been used to assess first-year students’ knowledge of design and their 
application of the design process. These methods include surveys, interviews, talk aloud protocols, 
concept maps, written and oral reports, exams, as well as the evaluation of students’ final design 
prototypes [49] [50] [51] [52]. As seen in these papers, measuring changes in students’ acquisition 
and application of design process knowledge requires considerable time and resources. A method 
developed by Bailey et al. is faster and less resource-intensive than many cited above and was 
found to be reliable in identifying gaps in students’ understanding of the design process [53] [54]. 

In addition to identifying appropriate methods to measure changes in design process knowledge, 
it is important to evaluate the effectiveness of the flipped classroom model in a first-year engineering 
design course. Specifically, we are interested in finding out whether shifting the delivery of design 
process knowledge from a lecture model to a flipped model affects students’ ability to apply the 
design process. To answer this research question, three major strands of assessment data are being 
evaluated: 

1. Pre-and post-testing of students’ knowledge and application of the design process as measured 
by their critiques of a Gantt chart laying out a 14-week design process. 

2. Individual scores on exams that cover the engineering design process and key professional skills. 

3. Technical memos on establishing design criteria, brainstorming solution ideas, and applying 
Pugh matrices for evaluation. 

For this paper, we report results from the first and second assessment strands used to compare the 
lecture model to the partially flipped model. Data from fall 2013 and spring 2014 serve as the ‘control’ 
(i.e., lecture model) in this study, since it represents student performance during the most recent non- 
flipped implementation of ENG1120. Data from fall 2014, the first semester in which four design process 
topics were flipped, capture the performance of students participating in a partially flipped model. Only 
data from students who signed university-approved IRBs are included in the analysis. 

Previously published work by Bailey et al. and the authors have documented the development 
of the first assessment tool, where students critique a Gantt chart that lays out a 14-week schedule 
for an engineering design project (Figure 4) [44] [53] [54]. The pre-test prompt was administered 
during the first week of class as a take-home assignment before introducing students to the over¬ 
all design process. A brief explanation of Gantt charts was provided to aid students with no prior 
exposure to them. The post-test was administered as a take-home assignment during the final exam 
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Project Prompt : The Engineers Without Borders (EWB) team from Rice University has recently 
completed a small health clinic to serve a rural town in Nicaragua. At this time, the clinic does 
not contain examination tables, which are necessary as many proper physical examinations 
and some treatments require that a person lie down. Thus, the goal of this project is to develop 
and build an examination bed for the EWB-built clinic in Nicaragua. 

Assignment : Critique the proposed 14-week design process to create an examination bed for the 
clinic in Nicaragua. This process is displayed in the Gantt chart below. 

• Elaborate on the steps in the design process with specific details. 

• Elaborate on specific strategies appropriate to accomplish the steps in the design process. 

• Identify the pros (advantages, strengths, etc.) and cons (disadvantages, weaknesses, etc.) of 
the proposed design process. 

Note that no work on this project was done prior to what is shown in the chart. 



Week 

Activity: 

1 

2 

3 

4 

5 

6 

7 

8 

9 

10 

11 

12 

13 

14 

Create many different 
concepts through 
brainstorming 















Based on needs, select the 
most promising concept 















Build prototype 















Test the prototype to ensure 
needs are met 















Make revisions to design 
based on test results 















Build final design 















Documentation 
















Figure 4. Pre- and post-test assessment prompt. 


period. Responses to the prompt were recorded in an online course management system and varied 
in length from one-half to three pages. 

In preparation for data analysis, all identifying information (name, date, pre- vs post-) was 
stripped from the pre- and post-test responses, and they were randomized. Trained raters evalu¬ 
ated the responses using a scoring rubric that was adapted from Bailey and Szabo’s studies [53] 
[54]. Eight topics, called Levels, were scored on a three-point scale (0,1, 2); six are relevant to steps 
in the design process and are reported here (Table 4). (Note that these Levels do not correspond 
with the levels in Bloom’s taxonomy.) An explanation of the Gantt chart exercise Levels and scoring 
strategies are explained elsewhere [44]. As noted in the third column of Table 4, three Levels were 
entirely flipped and one Level had some flipped components during the fall 2014 implementation. 
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Level 

Topic 

Flipped in 
Fall 2014? 

1 

Needs assessment and establishing design criteria 

Yes 

2 

Design context review 

No 

3 

Idea generation/Brainstorming 

Yes 

4 

Analysis and decision-making 

Yes 

5 

Building and testing 

No 

6 

Overall layout of a design process and iteration 

Some 


Table 4. Levels for coding responses to Gantt chart assessment. 


Results across the six Levels for the lecture (n = 78 for F13, S14) and partially flipped (n = 75 for 
F15) models are shown in Table 5. Scores range from 0-2, with 0 being low and 2 being high [44]. 
At the start of the class, student knowledge is low (<1.0) for needs assessment/establishing design 
criteria, design context review, and analysis and decision-making. At the end of the semester, values 
are high (all >1.0; a majority >1.5). 

Because the data are ordered categorical and paired, a generalized McNemar’s test was used to 
measure statistically significant differences between pre-test and post-test values [55]. For each Level, 
student responses were summarized in a 3x3 table (each table side as 0, 1, 2). This analysis directly 
incorporates the pairing of the pre- and post-test data, as each student’s pre-test and post-test value 
results in one value scored. Thus trends for pre-test and post-test responses can be seen, as well as 


Lecture (F13, S14): Pre-test 

Level 1 

Level 2 

Level 3 

Level 4 

Level 5 

Level 6 

Mean 

0.45 

0.53 

1.13 

0.69 

1.65 

1.28 

Standard deviation 

0.60 

0.73 

0.61 

0.54 

0.48 

0.56 

Lecture (F13, S14): Post-test 

Level 1 

Level 2 

Level 3 

Level 4 

Level 5 

Level 6 

Mean 

1.42 

1.34 

1.51 

1.78 

1.75 

1.58 

Standard deviation 

0.67 

0.80 

0.57 

0.56 

0.46 

0.49 

Partially Flipped (F14): Pre-test 

Level 1 

Level 2 

Level 3 

Level 4 

Level 5 

Level 6 

Mean 

0.23 

0.29 

0.95 

0.58 

1.58 

1.21 

Standard deviation 

0.49 

0.58 

0.64 

0.56 

0.49 

0.53 

Partially Flipped (F14): Post-test 

Level 1 

Level 2 

Level 3 

Level 4 

Level 5 

Level 6 

Mean 

1.21 

1.09 

1.53 

1.75 

1.78 

1.51 

Standard deviation 

0.81 

0.86 

0.57 

0.58 

0.41 

0.53 


Table 5. Assessment of design process knowledge using lecture and partially flipped models. 
Lecture data collected in fall 2013 and spring 2014; partially flipped data collected in fall 2014. 
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Level 

Topic 

P value: Lecture 
Pre vs Post 
(F13, S14) 

P value: 

Partially Flipped 
Pre vs Post (F14) 

1 

Needs assessment and establishing design criteria 

<0.001 

<0.001 

2 

Design context review 

<0.001 

<0.001 

3 

Idea generation/Brainstorming 

<0.001 

<0.001 

4 

Analysis and decision-making 

<0.001 

<0.001 

5 

Building and testing 

<0.1 

<0.001 

6 

Overall layout of a design process and iteration 

<0.001 

<0.001 


Table 6. Comparison of pre-test values versus post-test values for Gantt chart 
assessment. Data is reported as P values from the McNemar’s test for different class 
models: lecture (fall 2013 > spring 2014) and partially flipped (fall 2014). 


changes in student responses for each Level. The test statistic and P value at each Level were computed 
from the 3x3 table. 

As shown in Table 6, there are statistically significant differences between pre-test and post-test 
values for Levels 1-4 and 6. These results are consistent with prior work [56]. Results from Level 5 
for the partially flipped model, but not the lecture model, show statistically significant differences 
between pre- and post-test values. Overall these results indicate that students who take ENGI 120 
become capable of critically analyzing a proposed design process with high proficiency. 

With this as a foundation, we are primarily interested in comparing the results from the lecture 
model with the partially flipped model. Are there differences in students’ application of the engineer¬ 
ing design process when comparing delivery of engineering design material using a lecture model 
versus a partially flipped classroom model? The generalized McNemar’s test cannot be used to 
answer this question because it relies on paired data from each student. As an alternate to handling 
non-normal data, a permutation test was conducted to check for differences [55]. 

The permutation test was evaluated for three comparisons between the lecture and partially 
flipped models: differences between unpaired proportions on the pre-test values, differences be¬ 
tween unpaired proportions on the post-test values, and differences of changes between unpaired 
proportions on the pre-test and post-test values (Table 7). 

As shown in Table 7, pre-test values for most Levels for the partially flipped model are the same or 
lower as compared to the lecture model. No correction was made to adjust for these differences. With 
the permutation test, we see statistically significant differences for only Level 1 (P<0.02) and Level 2 
(P<0.05) for pre-test values, with partially flipped being significantly lower than lecture. The authors 
hypothesize that lower values (e.g., Level 2: 0.53 for lecture vs. 0.29 for partially flipped, see Table 5) 
may be due to the absence of any spring semester students, who may have been exposed during the 
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Level 

Topic 

P value: Lecture (F13, S14) vs Partially Flipped (F14) 

Pre-test Post-test Difference of Post-test and Pre-test 

1 

Needs assessment and establishing design criteria 

<0.02 

<0.1 

>0.9 

2 

Design context review 

<0.05 

<0.1 

>0.9 

3 

Idea generation/Brainstorming 

<0.1 

>0.9 

>0.1 

4 

Analysis and decision-making 

>0.2 

>0.5 

>0.4 

5 

Building and testing 

>0.3 

>0.5 

>0.3 

6 

Overall layout of a design process and iteration 

>0.4 

>0.2 

>0.9 


Table 7. Comparison of the lecture and partially flipped models for pre-test values, 
post-test values, and difference between pre-test and post-test values. Data is reported 
as P values from the permutation test. 


preceding fall to engineering design through other classes and clubs. When spring 2015 data is added, 
this difference may disappear. No statistical differences are seen for post-test values across all six Levels. 

The difference between the post-test and pre-test measures the “gain” in student learning, which 
is shown in the last column of Table 7. As you can see, there is no difference in this “gain” when com¬ 
paring the lecture model with the partially flipped model. In other words, the changes across the 
semester when comparing lecture versus partially flipped are not statistically significant for Levels 
1-6. For example, consider Level 1 in Table 5. Regardless of any difference in Level 1 pre-test values, 
the difference between post-tests and pre-tests for lecture and partially flipped model is -1.0. In sum¬ 
mary, there seems to be no difference in student performance for the listed topics when comparing 
the lecture and partially flipped methods. 

Levels 1, 3 and 4 explicitly target material that was taught using a partially flipped model in fall 2014. 
For these two topics, the difference or “gain” between the pre-test and post-test is larger for the partially 
flipped model than for the lecture model. Specifically for Level 3, the difference is 0.38 for the lecture 
model and 0.58 for the partially flipped model. For Level 4, the difference is 1.09 for the lecture model 
and 1.16 for the partially flipped model. However, there are no statistically significant differences. 

The second strand of assessment involves comparing exam grades for students in the lecture and 
partially flipped models. In the lecture model (n = 132 for F13, S14), the average exam score was 83.5 + 9.2. 
In the partially flipped model (n = 96 for F14), the average exam score is 83.0 + 7.0. Using a student’s 
t-test, there is no difference in the grades across the lecture and partially flipped models (P>0.5). 

To date, we have used two threads of assessment, the Gantt chart prompt and the exam. Both 
methods showed no statistically significant differences in students’ application of the engineering 
design process when comparing the lecture model with the partially flipped model. At minimum, 
we can say that no harm has been done by adopting the flipped classroom model. 
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DISCUSSION 

Many engineering faculty are motivated to flip their traditional lecture-based courses to promote 
active learning [13] [19] [25] [26] [27]. This was not the case in ENGI 120, a project-based design 
course in which 50% of class time was already devoted to active learning prior to flipping the course. 
Our goal in flipping ENG1120 was instead to replace the in-class lectures on the design process with 
activities explicitly designed to target higher levels of Bloom’s taxonomy and to engage students 
more fully in practicing the steps of the design process. When ENGI 120 conformed to the lecture 
model, students often operated at the level of remembering and understanding concept knowledge 
prior to tackling their own design project. In the flipped model, the ICEs prompt students to apply 
their knowledge of design concepts and to analyze and evaluate outcomes. We hypothesized that 
this practice would enable teams to approach their projects in more sophisticated, systematic ways. 
However, the study design did not permit us to directly assess this hypothesis. 

Assessment 

Assessment compared the lecture model used in fall 2013 and spring 2014 with the partially 
flipped model used in fall 2014. Recall that in fall 2014, four of the eight lecture periods on the de¬ 
sign process were flipped. Results from the two assessment methods show no difference in student 
performance between the lecture and partially flipped models. This neutral effect is consistent with 
much of the work on flipped engineering courses [13] [26] [28] [29] [30] [31] [38] [40] . In the 
case of ENGI 120, it was probably unrealistic to expect statistically significant changes in learning 
outcomes, since it has been an inquiry-based course since its inception [7], and only half of the 
design process lectures were flipped at this assessment point. 

While some studies have shown improvement in student performance in a flipped vs. traditional 
class, these studies focus on content-based courses, such as control systems in mechanical engi¬ 
neering, introductory circuits, pharmaceutics, or introductory statistics [25] [41] [42] [57] [58]. 
In contrast, we are teaching an engineering design/build course and evaluating improvements in 
process-based knowledge. In this way, our implementation is unique and may help lead the field 
in flipped classroom development and assessment for inquiry-based and process-based courses. 

Limitations 

For this project, we thought carefully about assessment in the context of the resources we have 
available. The two assessment methods deployed directly measured student learning and were 
summative in nature. Specifically, the Gantt chart assessment and the exam evaluate integrated 
knowledge learned through the lecture or video/ICEs, in addition to completion of a semester-long 
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design project. As a result, it is difficult to tease out the effects of the flipped model on students’ 
application of the design process. Neither of the methods directly measure how the use of flipped 
resources and in-class activities helped or hindered students as they tackled their own design chal¬ 
lenges. Specifically, we were not able to measure if this approach enabled teams to tackle their 
projects in an accelerated or more sophisticated way. 

Formative assessment of design knowledge is needed to thoroughly understand the impact of 
flipping this first-year course. Our efforts suggest that the development of automated, reliable, and 
accurate assessment tools designed to support the coding of qualitative interview data, open-ended 
survey responses, team documents, as well as video recordings of team meetings and oral presen¬ 
tations would allow educators to evaluate design process knowledge as students learn and apply 
the material directly to a project. This task is made more challenging, given that different teams are 
often working on different projects. 

Another limitation to our assessment is that we did not attempt to measure student motivation 
or satisfaction with the flipped model. While ENG1120 students have responded favorably, no data 
were collected to support or refute this claim. 

Reflections on Flipping a Classroom 

As this project nears its completion, the team does have several tips that may assist faculty who 
are considering such an undertaking. First, the process of flipping a classroom can be a fantastic 
opportunity to rethink and revise the content and organization of material delivered to the students. 
Second, to produce high-quality materials, we found that the whole process, from rethinking content 
to a final, professionally-edited video, took 10-20 hours of instructor time for each 30-45 minutes of 
traditional lecture. We recommend partnering with an institution’s technology team, where possible, 
to facilitate this process. Working with a project manager and a video editor who can understand 
both the technical content and the pedagogical purpose of the material can also accelerate the 
production and refinement of the material. Third, it is important to be familiar with a range of active 
learning strategies that can be implemented in class to replace the lecture time. Finally, carefully- 
crafted assessment tools that meet the instructor/researcher’s needs must be thoughtfully prepared 
and deployed. Depending on the student learning outcomes for the course, formative and summative 
assessment can be fairly straightforward or intensely challenging. 


FUTURE WORK 

The team’s long-term future plans are to expand the on-line, open access repository of rich in¬ 
structional materials we have developed to support the transition to a fully flipped classroom model 
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in engineering design courses. Production of all the videos, quizzes, and in-class exercises has been 
completed (Tables 2 and 3). We hope other institutions will use materials from this repository to 
augment their course materials and contribute materials to the repository for others’ use. While 
we anticipate some challenges related to nomenclature and differences in design process models, 
these can be overcome through coordination. We believe that such a repository of rich media will 
improve first-year engineering design education across the country. 

The team has continued its assessment work. By fall 2015, all eight design process lectures and 
six (of eight) professional skills lectures were fully flipped. The summative assessment of the Gantt 
chart exercise and the exam are being analyzed for this fully flipped model and compared to the 
control lecture case. 

We are analyzing the third strand of proposed assessment, specifically the first drafts of ENGI 
120 teams’ technical memos. These memos isolate specific steps in the design process, such as 
establishing design criteria, brainstorming solution ideas, or applying Pugh matrices for evaluation. 
This analysis may provide a more formative view of students’ knowledge. That said, the memos 
represent a team effort not an individual effort, and they also reflect interactions with teaching 
assistants, writing mentors, and the instructors. These factors complicate our ability to attribute 
learning outcomes to flipped pedagogical practices. 

Additionally, the evaluation of technical memos cannot assess how the teams initially tackled 
their project, specifically whether the teams moved more fluidly and competently through the steps 
in the engineering design process. We want to encourage the engineering education community to 
expand the research and evaluation tools for the formative assessment of engineering design process 
knowledge. Ideally, these tools would enable us to determine whether a flipped model accelerates 
the design process and enables the students to create more sophisticated, functional prototypes. 

The team is also considering other strands of evaluation, specifically engineering identity for¬ 
mation. The team has access to student self-report surveys that explore student’s self-concept, 
self-efficacy, and commitment to engineering. However, we remain doubtful that significant differ¬ 
ences between a lecture and a flipped model will be apparent, as we hypothesize that it is student 
involvement with the client-based project that is central to the formation of engineering identity. 


CONCLUSION 


In summary, faculty at Rice University developed instructional resources to teach first-year engi¬ 
neering design using a flipped classroom model. By employing the flipped model, lecture time was 
replaced with videos and in-class exercises that highlighted important steps in the engineering design 
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process and provided opportunities for students to practice them. All of these materials are available 
in a web-based repository. Two direct assessments of student learning comparing the partially flipped 
model to the lecture model showed no statistically significant differences. Measuring the impact of 
flipping a project-based design course is challenging because significant active learning is already 
embedded in this type of course. 
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APPENDIX A 

Quizzes and in-class exercises are shown for the screening Pugh matrix module. For the quiz, 
four example questions are shown. Most test whether students remember or understand the 
material. For the ICEs, two examples are shown. These ICEs require students to analyze and 
evaluate material. The ICE #1 illustrates common, difficult decisions, whereas ICE #2 shows two 
common errors. 

Screening Pugh Matrix Quiz 

1. The purpose of a Pugh screening matrix is to - 

A. determine the best solution. 

B. increase the number of solution ideas. 

C. recombine solution ideas to form more complete ideas. 

D. reduce a large number of ideas to a few ideas. 

2. Decision matrices are evaluated using - 

A. a formula. 

B. pre-existing criteria. 

C. engineering software. 

D. iterative process diagrams. 

3. What three steps should be used to narrow down design ideas? 

A. Scoring matrix 2. Screening matrix 3. Common sense 

B. Screening matrix 2. Common sense 3. Scoring matrix 

C. Common sense 2. Screening matrix 3. Scoring matrix 

D. Screening matrix 2. Scoring matrix 3. Common sense 

4. Which of the following can contribute to ineffective engineering decision matrices? 

A. Poor choice of standard solution 

B. Nonsensical design ideas 

C. Poor choice of design criteria 

D. A and C 
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Screening Pugh Matrix: In-Class Exercise #1 

Your design team is tasked with building an 
improved hay feeder for the giraffes at your local 
zoo. In talking with the keepers, you identify five 
design objectives: 

• Safety 

• Extended feeding time 

• Food capacity 

• Durability 

• Blends in with exhibit 

Your team has brainstormed 75 design so¬ 
lution ideas and needs to use a scoring Pugh 
matrix to down-select the number of ideas. Your team had decomposed its task into several design 
blocks, including “methods for delivering hay.” Six brainstormed ideas that fall into this design block 
are as follows: 

• Holes in a plastic surface 

• Holes in a metal surface 

• Metal grating (similar to existing feeder, shown at right) 

• Canvas straps placed adjacent to one another 

• Wire mesh 

• Food delivered in small diameter tubes 

Your team has also collected information on the following brainstormed ideas: 

1. Metal surfaces are more likely to harm giraffes, because of sharp edges. 

2. Metal is known to be durable for 10 yrs. Plastics are durable for 5 yrs. Solid surfaces are 
more durable than mesh or other materials with many holes. Canvas is similar in durability 
to plastic. 

3. To blend in with an exhibit, the feeder should look like a tree. Color is important, and 
materials that are naturally brown, green or tan or can be painted one of those colors is 
more desirable. Metals that are painted retain their color much more readily than plastics 
that are painted. 

4. To really extend feeding time, the surface area of the exposed hay must be less than 1-2 ft 2 . 

Develop a screening Pugh matrix to evaluate the design block of “methods for delivery of hay” 

using appropriate design criteria. It is recommended that your team sketch out and discuss the 
features of the solution ideas look like BEFORE starting a Pugh matrix. This way, you will be talking 
about the same ideas... 
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What solution idea served as the standard? Which solution ideas should be moved on for further 
consideration? Which solution ideas should be dropped? 

Pugh Screening Matrix In-Class Exercise #2 

An engineering team generated the following screening Pugh matrix. Critique the matrix carefully. 



Idea A 

Idea B 

Idea C 

Idea D 

Idea E 

Durability 

0 

+ 

+ 

- 

0 

Ease of use 

0 

- 

0 

- 

0 

Cost 

0 

0 

+ 

0 

- 

Portability 

+ 

+ 

+ 

1 

+ 

Sum 

+1 

+1 

+1 

-1 

0 
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